Resume host access based on transaction logs

ABSTRACT

Examples disclosed herein relate to processing transaction logs for storage nodes. Multiple nodes manage transactions for storage units including transaction logs. A node assumes control over a set of storage units and blocks host access requests to the storage units. Transaction logs are processed to determine a subset of the set to perform recovery for. In this example, the subset of storage units are locked. Host access requests are resumed.

BACKGROUND

Some distributed file systems heretofore use a network of controllers tomanage transactions to and from a storage unit. Such controllers mayreplicate transaction logs amongst themselves. In the event a controllerfails, the transaction logs may be used to determine the transactionsthat were in progress when the controller failed.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description references the drawings, wherein:

FIGS. 1 and 2 are block diagrams of computing systems capable ofresuming host access requests after a node takes over control of storageunits in a redundant system, according to various examples;

FIG. 3 is a flowchart of a method for using temporary meta-data datastructures to expedite resumption of host access requests after a nodetakes over control of storage units in a redundant system, according toan example;

FIG. 4 is a block diagram of a computing device capable of usingtemporary meta-data data structures to expedite resumption of hostaccess requests after a node takes over control of storage units in aredundant system, according to an example;

FIG. 5 is a flowchart of a method for paging in an exception tableduring recovery, according to an example; and

FIG. 6 is a flowchart of a method for prioritizing paging in of anexception table based on a request from a host, according to an example.

DETAILED DESCRIPTION

A computing device may store data in a file system, which may store datain files, directories, or other “file system objects”, and may storevarious meta-data associated with each file system object. In someexamples, computing device(s) may store data in a distributed filesystem (DFS) providing global file system namespace semantics for thestorage of the DFS, where different portions of storage of the overallDFS are physically separate from one another, logically separate fromone another, or a combination thereof, though accessible to globalfunctionalities of the DFS (e.g., via at least one computer network).

In some examples, a DFS may include a plurality of storage nodes or“nodes,” each to manage one or more storage units of the DFS. In someexamples, the storage nodes may be physically or logically separate orremote from one another (or a combination thereof) but accessible to oneanother or to at least one storage node implementing global functionsfor the DFS (e.g., via at least one computer network). In examplesdescribed herein, a storage node may comprise at least one computingdevice to implement the functionalities of the storage node, includinglocally managing one or more storage units of the DFS. In otherexamples, storage nodes may be implemented by a combination of separatephysical computing devices and different logical computing devices(e.g., virtual machines or the like).

In examples described herein, a “storage unit” may be a discretephysical and/or logical portion of storage in a DFS that implements itsown independent local file system specific to that portion of storage.In some examples, different storage units may be implemented bydifferent storage devices (e.g., different physical disks, drives,etc.), by different portions of a single storage device, or acombination thereof. In some examples, a storage unit managed by anassociated storage node may be implemented by at least one storagedevice physically connected locally to a computing device implementingthe storage node, or may be implemented by a portion of a storage array(e.g., a given logical unit of storage identified by a given logicalunit number (LUN)).

In examples described herein, meta-data is a set of data that givesinformation about other data. In the context of a storage system,examples of meta-data include a bitmap, exception tables, log pages,etc.

In one example, a bitmap is used for space management. When the storagesystem needs space (e.g., for meta-data like an exception table or fordata that a host wants to write), the storage system goes through thebitmap list that belongs to that LUN to determine where there is freespace and allocate needed space. In some examples, finding free spacemay include looking through information that is not stored in a mainmemory of the system node that is looking.

In another example, a “log page” is for a space freeing operation. Thelog page stores locations where free operation needs to be performed.Consider an example in which LUN 20, offset 120 was originally pointingto logical disk (LD) 1, offset 10, and later, a write full off zerosoccurs to LUN 20, offset 120. If the system has a special representationof zero, space is not needed for storing the zero data. As such, LD 1,offset 10 can be freed. Instead of updating bitmap directly, LD 1,offset 10 can be put into the log page, which eventually can beprocessed by a background thread. In some examples, a transaction logentry can be made for the updating of the log page until it is flushedto a non-volatile memory.

As used herein, the term “exception table” is a data structure that caninclude virtual volume to logical drive mapping information. Forexample, the exception table can be made up of address tables that mapvirtual volume pages to storage units (e.g., logical disk pages). Asstorage demand grows, additional storage is allocated for the addresstables and the data pages from separate pools of storage. If any of thepools runs low, more logical disk regions are allocated to that pool. Assuch, hosts can be provided access to storage space that can bevariable. Further, multiple levels of exception tables can be used todetermine a logical/physical location for a storage unit. For example, afirst level of exception table can be pointed to and known to each nodein the system. The exception table entries can reference a second levelof exception tables. Similarly, further layers of exception tables canbe used. In one example, a virtual identifier and offset can be used todetermine from the first level exception table a second level exceptiontable. Then, the virtual identifier and offset can be used to determinefrom the second level exception table a mapping to a physical storageunit or another level of exception table. Due to large amounts ofstorage that can be controlled, the size of exception tables and/orother meta-data data structures can be large e.g., in the order ofhundreds of megabytes or larger.

Storage nodes can act as a cache (e.g., to write information) for thestorage units. Hosts can request to read from and/or write to thestorage units using the storage nodes. Storage nodes in a network mayreplicate transaction logs amongst themselves for fault tolerancepurposes. If one node fails, another storage node may substitute for thefailed storage node. By analyzing transaction logs, the substitutingstorage node may continue from where the failed storage node ended.

Moreover, in a fault tolerant system, a copy of transaction logs and acopy of the cached data (e.g., information to be written) can be kept attwo or more nodes. In some examples, the cached data can be implementedas cache memory pages. Further, the write information can include cachememory pages that are marked as dirty (or altered). As such, a storagenode that controls a set of storage units can include, in a memory onthe storage node, write information as well as transaction logs. Thecontrolling storage node can also copy the transaction logs to anothernode. Further, the controlling storage node can also copy the writeinformation to another node. Eventually, the write information will beflushed to the corresponding storage units. Moreover, the node with acopy of the transaction logs and the node with the copy of the writeinformation can be the same node. In the fault tolerant system, a goalcan be to provide the hosts with at least two copies of the transactionlogs and write information when providing host read and write access. Assuch, when the controlling storage node fails, the storage nodereceiving control processes the transaction logs to ensure that twocopies of the write information and transaction logs exist.

As used herein, the term “transaction log” is a log that includes alocation of two copies of data (e.g., the write information) on twonodes, updates to meta-data (e.g., a new exception table entry value),and information about where the data is to be written (e.g., thelocation of the storage unit, such as a LUN, offset that the host wroteto). When a host writes to a storage unit, the controlling node createsthe transaction log and a copy of the transaction log is made at asecond node. Further, the write information is included on thecontrolling node and a copy of the write information is kept at a secondnode. The copy of the transaction log and the copy of the writeinformation can be on the same node or can be on different nodes.

In one example, during normal operations, an invariant property that thesystem can provide can be that there are copies of write information andtransaction logs when host access is provided to a storage unit. Whenone node dies, such invariant no longer holds. In this example, the term“playback” is the operation that traverses through transaction logs andcreate new copies of write information and transaction logs, so that theinvariant still holds after playback finishes. There is no assumptionthat the node that died will come back.

During processing, the input/output (IO) from hosts to the storage unitsin the set are blocked to ensure fault tolerance. However, this may taketime because some of the information used to complete recoveryoperations associated with the transaction logs may be stored on one ofthe storage units, which takes more time to page in compared to, forexample, main memory of a storage node. As used herein, “main memory” ofa node is memory that is directly accessible by a processor of the node.During the time that the host access to IO is blocked, the IO stall timemay cause application failures.

Accordingly, various embodiments disclosed herein relate to expeditingthe amount of time it takes to enable host IO to resume after a storagenode fails and its associated storage unit set controlled by a newstorage node. As used herein, a storage node that “fails” or dies is astorage node that will no longer control a set of storage units that itis controlling. This can be, for example, due to an unexpected failure,a node going down to restart, or transferring for some other reason.

When a storage node receiving control for a set of storage unitsreceives control, receiving storage node blocks host access requests tothe set of storage units transferred. The receiving storage nodeprocesses the transaction logs to determine a subset of the set of thestorage units that it is necessary to perform recovery on, for example,because those storage units have pending activity that has not beencompleted. Some recovery operations can be quickly performed by thereceiving storage node while other recovery operations may take anextended period of time, for example, when an exception table or othermeta-data needs to be loaded from a storage unit to perform recovery.

As such, in the example of paging in an exception table, a temporaryexception table can be created instead of loading the exception table.Similarly, a corresponding temporary meta-data data structure can becreated for other meta-data that may need to be paged in. The temporaryexception table or temporary meta-data data structure would take time onthe order of a memory allocation. Loading of the exception table can bequeued in the background while the temporary exception table can be usedto flag that the storage unit has one or more recovery operation pendingand lock the storage unit from being accessed by a host. The temporaryexception table can also include the associated recovery transaction ortransactions. In some examples, the recovery transaction or operationincludes a change to be made to the exception table. For example,suppose exception table A, entry 10 was empty, then host wrote to thelocation and a mapping to logical disk LD, offset 100, exception table Aentry 10 will be updated to have this information after write completes.If node went down in between, then during recovery the system wouldcreate a recovery item denoting that entry 10 is to be updated withmapping to logical disk LD, offset 100. As used herein, virtual volumesand virtual volume pages can be referred to as virtual storage unitswhile physical disks or drives such as logical disks can be referred toas storage units. The host IO requests can reference the virtual storageunits, for example, by using a virtual identifier and offset. Thisvirtual identifier and offset can be used to look up in a first levelexception table, a second level exception table (similarly, the secondlevel exception table can be used to look up a third, etc.). If a levelof the exception tables are not in a main memory of the node receivingcontrol, a temporary exception table can be made and referenced in adata structure (e.g., a hash table). That way if a host IO request comesin requesting the virtual storage unit, the node receiving control willknow that a recovery operation is pending on the virtual storage unitand, thus, the storage unit.

Once the temporary exception tables are made from the processedtransaction logs, the host access requests can be resumed. With thisapproach, just the storage units with pending recovery are locked frombeing accessible from the host access requests. Moreover, host accessrequests can resume for all of the storage units.

When a host access request is received, the receiving storage node cancheck its exception tables, including the temporary exception tables,using the data structure to determine whether the storage unit to beaccessed has a pending recovery transaction associated. If there is notpending recovery transaction, then the IO can be processed normally. Ifthere is a pending recovery transaction associated, the answer to thehost request can wait for paging in of any needed exception tables,which can be prioritized compared to other recovery transactions. Then,the lock on the storage unit can be released and the host request can beanswered using normal processes. In one example, the lock is thepresence of the temporary exception table and unlocking can be performedby removing the temporary exception table from the data structure.

The aspects, features and advantages of the present disclosure will beappreciated when considered with reference to the following descriptionof examples and accompanying figures. The following description does notlimit the application; rather, the scope of the disclosure is defined bythe appended claims and equivalents.

FIGS. 1 and 2 are block diagrams of computing systems capable ofresuming host access requests after a node takes over control of storageunits in a redundant system, according to various examples. In oneexample, system 100 can include a plurality of storage nodes or nodes102 a, 102 b-102 n that can be used to control storage units 104 a-104m. The nodes 102 and storage units 104 can communicate via one or morenetwork 106. In this example, node 102 a can control a set of thestorage units, for example, storage units 104 b-104 i (not shown) andinclude a transaction log 110. Further, the node 102 a can include writeinformation 120 from host requests that have not yet been written to thestorage units 104 b-104 i. Node 102 b can include a copy of thetransaction log, or transaction log copy 112 as well as a control engine130 to control access to storage units and a transaction engine 132 toprocess transaction logs.

Moreover, the example of FIG. 2 shows system 200, which further includeshosts 250 that request IO access to the storage units 104 from the nodes102 via a network 210. In this example, node 102 b further includes arecovery engine 234 and a page in engine 236 as well as at least oneprocessor 240 and memory 242. Moreover, node 102 n can include a copy ofthe write information 222.

As noted, node 102 a can control storage units 104 b-104 i. The node 102a can manage transactions to and from the storage units 104 b-104 i.Managing transactions can include maintaining transaction logs describedabove including details of the transactions we well as managingassociated write information. As noted above, the details of thetransactions maintained in the transaction logs include a location ofthe write information, a location of a copy of the write information (onanother node, for example, write information copy 222 on node 102 n),updates to exception tables, and information about where (e.g., whichstorage unit) the write information is to be written (e.g., in the formof a LUN and offset that the host wrote to). As shown, node 102 bincludes a transaction log copy 112, which is a copy of the transactionlogs 110. When node 102 a fails, the system 100, 200 can be configuredto have node 102 b assume control over the set of storage units 104b-104 i.

The control engine 130 can assume control over the set of storage units104 b-104 i. Though the set is described in the numerals 104 b-104 i,the set does not need to be continuous. At this point, the system 100,200 does not have at least two separate copies of the transaction logsand write information. As such, node 102 b begins recovery operations.Control engine 130 can block host access requests to the set of storageunits 104 b-104 i. Host requests can either return an error or be queueduntil processing resumes.

Transaction engine 132 processes the transaction logs to determine asubset of the set of the storage units to perform recovery for and toupdate a data structure such as a hash table or linked list to lock thesubset of the storage units. As used herein, a subset includes at leastone of the storage units from the set of storage units, but not all ofthe storage units from the set of storage units. In some examples, thedata structure includes the type of information being stored, forexample, a normal IO operation or a recovery operation. During normaloperation, the data structure can be used to point out which data is inthe cache. In some examples, the data structure, during this recovery,can be created or updated from the transaction logs. Processing caninclude playing back the transaction logs to create new copies of thewrite information and to create new transaction logs at another node orother nodes. In one example, the old transaction logs would have thelocation of the write information in node 102 a and 102 n. The newtransaction logs would have the location of the write information atnode 102 n and another location (e.g., node 102 b). In some examples,the node with the write information copy is actually node 102 b and thecopies of both the transaction logs and write information are on thesame node.

During playback of the transaction logs copy 112, the playback can besped up by using temporary meta-data data structures such as temporaryexception tables. The recovery engine 234 can initiate recoveryoperations for the storage units from the subset of the storage units(e.g., storage units 104 c-104 h). For each of the recovery operationsthat are associated with a data structure, such as an exception table,it can be determined whether the meta-data or exception table needs tobe paged in from a slower memory source than main memory of the node 102b or another one of the nodes 102 connected via a fast link (e.g.,non-main-memory or a non-volatile memory). If so, a temporary meta-datadata structure such as a temporary exception table is created andincludes an associated recovery transaction (e.g., marking, in anexception table, a physical storage unit to be associated with a virtualvolume in the transaction log). The temporary meta-data data structures,such as temporary exception tables can be created before host accessrequests are resumed.

As noted above, part of the recovery operations may include paging in ofan exception table from a storage unit (e.g., a unit that is not mainmemory of the node 102 b). A data structure, such as a hash table can beused to control exception tables and other temporary meta-data datastructures (e.g., temporary exception tables). In some examples, thedata structure can distinguish between normal IO transactions andrecovery related IO transactions. For example, the exception tables andtemporary exception tables can have a flag that distinguish them as theexception table or a temporary exception table. Storage units associatedwith the temporary exception tables can be considered implicitly locked.In other examples, an explicit lock can be used.

Once the temporary exception tables are in place, the control engine 130can resume host access requests. In one example, a data structure canidentify each of the temporary exception tables. Associated exceptiontables can be paged in in the background, for example, once the hostaccess requests have been resumed. As such, the page in engine 236 canpage in the exception tables from the slower memory to a main orvolatile memory of the node 102 b. As noted above, the exception tablescan include mapping information for the write information. Resuming hostaccess requests can mean that the host access requests are unblocked andare serviced and processed.

As noted above, the write information copy 222 can be caused to becopied by the recovery engine 234 to another node, for example, node 102b. Portions of the write information can be copied at a time as part ofthe processing of the transaction logs.

Once IO is resumed, the control engine 130 can receive requests fromrespective hosts 250 over network 210. The control engine 130 canreceive a host request for storage unit 104 c, which is part of thesubset in this example. The control engine 130 can determine that thestorage unit 104 c is locked based on the hash table, based on lookingat the temporary exception tables, or using a similar lookup. Based onthe determination that the storage unit 104 c is locked, performance ofthe recovery transaction(s) associated with the storage unit 104 c areprioritized. The storage unit 104 c can be unlocked for the host requestonce the recovery transaction is completed. Meanwhile, other IO tostorage units from the set that were not locked can continue whilerecovery operations occur in the background. While waiting for therecovery transaction(s) for the storage unit 104 c to complete, hostaccess requests for the storage unit 104 c can be paused.

Though the example above is related to exception tables and temporaryexception tables. Similar approaches can be used when other meta-data(e.g., a bitmap or log page) is determined not to be located in a mainmemory of the controlling node 102 b and required to be paged in from aslower memory source.

In certain examples, nodes 102 are computing devices, such as servers,client computers, desktop computers, mobile computers, etc. In otherembodiments, the nodes 102 can include special purpose machines. Thenodes 102 can be implemented via a processing element, memory, and/orother components.

The engines 130, 132, 234, 236 include hardware and/or combinations ofhardware and programming to perform functions provided herein. Moreover,in some examples, modules (not shown) can include programing functionsand/or combinations of programming functions to be executed by hardwareto perform the functionality of the engines 130, 234, 236. Whendiscussing the engines and modules, it is noted that functionalityattributed to an engine can also be attributed to corresponding modules.Moreover, functionality attributed to a particular module and/or enginemay also be implemented using another module and/or engine.

A processor 240, such as a central processing unit (CPU) or amicroprocessor suitable for retrieval and execution of instructionsand/or electronic circuits can be configured to perform thefunctionality of any of the engines 130, 132, 234, 236 described herein.In certain scenarios, instructions and/or other information, such asexception tables, temporary exception tables, hash tables, etc., can beincluded in memory 242 or other memory. Moreover, in certain examples,some components can be utilized to implement functionality of othercomponents described herein. Input/output devices such as communicationdevices like network communication devices or wireless devices can alsobe included as components of the nodes.

Each of the engines may include, for example, hardware devices includingelectronic circuitry for implementing the functionality describedherein. In addition or as an alternative, each module correspondingmodule may be implemented as a series of instructions encoded on amachine-readable storage medium of a node and executable by processor.It should be noted that, in some embodiments, some modules areimplemented as hardware devices, while other modules are implemented asexecutable instructions.

Hosts 250, nodes 102, and storage units 104 may include networkinterface device(s) to communicate with other computing resource(s)(e.g., computing device(s)) via at least one computer network. Asdescribed herein, a computer network may include, for example, a localarea network (LAN), a virtual LAN (VLAN), a wireless local area network(WLAN), a virtual private network (VPN), the Internet, or the like, or acombination thereof. In another example, the storage units 104 may be astorage device residing on a storage network, such as a Small ComputerSystem Interface (“SCSI”) device presented to a Storage Area Network(“SAN”) using a Fibre Channel, Infiniband, or Internet Protocol (“IP”)interface. It is understood that each storage unit 104 a-104 m mayinclude any other type of storage unit and that the foregoing is anon-exhaustive list. In another example, storage units 104 may beconfigured as a volume that may be accessed via an operating system'slogical interface.

FIG. 3 is a flowchart of a method for using temporary meta-data datastructures to expedite resumption of host access requests after a nodetakes over control of storage units in a redundant system, according toan example. FIG. 4 is a block diagram of a computing device capable ofusing meta-data data structures such as temporary exception tables toexpedite resumption of host access requests after a node takes overcontrol of storage units in a redundant system, according to an example.The computing device 400 can be a node that takes over duties to processstorage units. The computing device 400 includes, for example, aprocessing element 410, and a machine-readable storage medium 420including instructions 422, 424, 426 for using temporary exceptiontables to expedite resumption of host access requests in a faulttolerant system. Computing device 400 may be implemented as, forexample, a notebook computer, a server, a workstation, or any othercomputing device.

Processing element 410 may be, one or multiple central processing unit(CPU), one or multiple semiconductor-based microprocessor, one ormultiple graphics processing unit (GPU), other hardware devices suitablefor retrieval and execution of instructions stored in machine-readablestorage medium 420, or combinations thereof. The processing element 410can be a physical device. Moreover, in one example, the processingelement 410 may include multiple cores on a chip, include multiple coresacross multiple chips, multiple cores across multiple devices (e.g., ifthe computing device 400 includes multiple node devices), orcombinations thereof. Processing element 410 may fetch, decode, andexecute instructions 422, 424, 426 to implement method 300. As analternative or in addition to retrieving and executing instructions,processing element 410 may include at least one integrated circuit (IC),other control logic, other electronic circuits, or combinations thereofthat include a number of electronic components for performing thefunctionality of instructions 422, 424, 426.

Machine-readable storage medium 420 may be any electronic, magnetic,optical, or other physical storage device that contains or storesexecutable instructions. Thus, machine-readable storage medium may be,for example, Random Access Memory (RAM), an Electrically ErasableProgrammable Read-Only Memory (EEPROM), a storage drive, a Compact DiscRead Only Memory (CD-ROM), and the like. As such, the machine-readablestorage medium can be non-transitory. As described in detail herein,machine-readable storage medium 420 may be encoded with a series ofexecutable instructions for using temporary exception tables to expediterecovery of a fault tolerant storage system.

As noted above, a plurality of nodes (e.g., Node A-Node N) can be usedin a fault tolerant system capable of storing information on storageunits. Computing device 400 can be one of the nodes. The Nodes can be incharge of managing transactions to and from a number of storage units.Node A can have control of managing transactions to and from a set ofthe storage units. Moreover, Node A can include write information forthe set of the storage units and can maintain corresponding transactionlogs that include details of the transactions, such as the location ofthe write information, the location of a copy of the write information,updates needed to exception tables, and where the information is to bewritten. A second node (Node B), for example computing device 400, caninclude a copy of the transaction logs. The computing device 400 mayalso include the copy of the write information or the write informationcan be located on another one of the nodes. As noted above, Node A mayfail.

When Node A fails, at 304, the processing element 410 can executecontrol instructions 422 to control assume control of the set of storageunits that were controlled by Node A. In one example, the control canshift once a trigger condition or event occurs, for example, a lack ofcommunication from Node A for a preset amount of time, an indicationfrom Node A, etc.

At 306, the control instructions 422 can be executed to block hostaccess requests to the set of storage units. This can be done to ensurethat fault tolerance is present before allowing access to the storageunits. As noted above, the blocking can be temporary to process thetransaction logs and ensure fault tolerance.

At 308, the recovery instructions 424 are used to determine a subset ofthe set of the storage units to perform recovery for based on the copyof the transaction logs. The transaction logs can be processed to createa new set of transaction logs that point to the location of the copy ofthe write information as well as a new location for another copy of thewrite information on another node (Node C), the update to meta-data, andinformation about where the data is to be written. Recovery instructions424 can also be used to cause the copying of the write information toNode C. Copying information between nodes can occur quickly (e.g., inthe order of milliseconds), which may be more than a memory operation ina single node, but an order of magnitude faster than using anotherstorage outside of main memory, such as a non-volatile storage such as asolid state drive or array, a hard disk drive or array, or other similarnon-volatile memory.

During the processing of the transaction logs, a page in of an exceptiontable may be required to complete the playback of the log. To shortenthe time for recovery of the storage units in the set, at 310, therecovery instructions 424 can be used to create a temporary meta-datadata structure, such as a temporary exception table 428 for one or moreof the storage units from the subset. This can be based on adetermination that the storage unit(s) is associated with one of themeta-data, such as exception tables, that is not stored in a main memoryof the computing device 400. The temporary meta-data data structures canalso serve as a lock on the subset of storage units. As such, lockinstructions 426 can be executed to set the storage units as locked. Asnoted above, the lock can be implicit, for example, by marking thetemporary meta-data data structure (e.g., temporary exception tables428) as temporary or fake compared to normal exception tables used.Moreover, the fact that the temporary exception tables 428 are temporarycan denote that the associated storage units are in recovery. As notedabove, the temporary meta-data data structures such as the temporaryexception table(s) 428 can be associated with a recovery transaction oroperation or multiple recovery transactions or operations. Though theexample used here is for an exception table and temporary exceptiontable, the process used can also be used for other similar meta-datasuch as bitmaps and log pages using a temporary meta-data datastructure. The temporary meta-data data structure and/or a datastructure pointing to the temporary meta-data data structure canindicate that the temporary meta-data data structure is associated witha recovery operation or transaction rather than normal IO.

At 312, the control instructions 422 can be executed by processingelement 410 to resume host access requests for the set of storage unitsafter creating the temporary meta-data data structures. Thus, IOrequests to the storage units in the set can be resumed from the hostsbefore all recovery for the set is complete and two copies of the writeinformation and new transaction logs are present. However, in thebackground, the meta-data such as exception tables, bitmaps, log pages,etc. are being paged in and the storage units still in recovery arestill locked using the temporary meta-data data structures. A datastructure (e.g., a hash table, linked list, etc.) can be updated to lockand unlock the subset of storage units based on the transaction logs. Asnoted above, the lock can be implicit (e.g., existence of a temporaryexception table or temporary meta-data data structure referencing thestorage unit (e.g., using a virtual identifier and offset)) or explicit.

FIG. 5 is a flowchart of a method for paging in an exception tableduring recovery, according to an example. Although execution of method500 is described below with reference to computing device 400, othersuitable components for execution of method 500 can be utilized (e.g.,node 102 b). Additionally, the components for executing the method 500may be spread among multiple devices. Method 500 may be implemented inthe form of executable instructions stored on a machine-readable storagemedium, such as storage medium 420, and/or in the form of electroniccircuitry. Though this method 500 focuses in on one exception table, theprocess can be implemented for multiple exception tables serially or inparallel.

At 502, an exception table is paged in to main memory of the computingdevice 400 after host access requests are resumed. In this example, thetime the exception table is paged in is the time that the page in iscompleted. As such, the page in process can begin before host accessrequests are resumed by computing device 400. In one example, a datastructure (e.g., a linked list or table) can be used to identify each ofthe temporary exception tables and can be used to process the paging inand associated recovery transactions or operations.

At 504, the associated recovery transactions for the temporary exceptiontable are performed using recovery instructions 424. As noted above, inone example, the recovery operation can be a change that needs to bemade to the exception table. Moreover, the associated recoverytransaction(s) can be implemented as a background process. For example,suppose exception table A, entry 10 was empty, then host wrote thelocation and Node A allocates a mapping to logical disk LD, offset 120,exception table A entry 10 will be updated to have this informationafter write to the storage unit completes. If Node A went down inbetween, then during recovery, the computing device 400 would create arecovery item denoting that entry 10 needs to be updated with mapping tological disk LD, offset 120.

At 506, the data structure managing locks for the storage units isupdated using lock instructions 426. In one example, the update includesremoving the temporary exception table, which also removes locking ofthe storage unit. As such, the data structure is updated to removeindication that the storage unit has an associated recovery transactionpending. In another example, an explicit lock on the storage unit can beremove. In some examples, the lock on the storage unit may also includea larger storage container including the storage unit, for example, ifthe storage unit is LD offset 120, LD offset 120 can be locked or alarger portion of the LD may be locked to IO during recovery.

FIG. 6 is a flowchart of a method for prioritizing paging in of anexception table based on a request from a host, according to an example.Although execution of method 600 is described below with reference tocomputing device 400, other suitable components for execution of method600 can be utilized (e.g., node 102 b). Additionally, the components forexecuting the method 600 may be spread among multiple devices. Method600 may be implemented in the form of executable instructions stored ona machine-readable storage medium, such as storage medium 420, and/or inthe form of electronic circuitry. Though this method 600 focuses in onone storage unit, the process can be implemented for other storageunits.

At 602, the computing device 400 receives a host request from a host foraccess to a storage unit. Lock instructions 426 can be executed todetermine whether the storage unit is locked (604). For example, thepresence of a temporary exception table associated with the storage unitcan mean that the storage unit is locked.

If the storage unit is locked, at 606, the recovery instructions 424 canbe executed to prioritize paging in of an exception table associatedwith the storage unit into a main memory of the computing device 400.Further the associated recovery transaction can be performed (608). Asnoted above, the associated recovery transaction can be an update of theexception table that is paged in. In some examples, when the exceptiontable is updated, the exception table can be marked dirty and laterflushed to non-volatile storage that the exception table was paged infrom. In other examples, the associated recovery transaction may alsoinclude causing a portion of a copy of the write information to becopied from one node to another to ensure fault tolerance. At 610, thestorage unit can be unlocked. As noted above, the unlocking can beperformed as an update to a data structure. Then, the host request canbe performed.

All of the features disclosed in this specification (including anyaccompanying claims, abstract and drawings), and/or all of the elementsof any method or process so disclosed, may be combined in anycombination, except combinations where at least some of such featuresand/or elements are mutually exclusive.

What is claimed is:
 1. A system comprising: a plurality of nodes formanaging transactions to and from a plurality of storage units and formaintaining transaction logs including details of the transactions,wherein a first one of the nodes controls a set of the storage units andincludes write information for the set of storage units and maintainscorresponding transaction logs for the write information, wherein asecond one of the nodes includes a copy of the transaction logs, thesecond one node further comprising: a control engine to assume controlover the set of storage units and to block host access requests to theset of storage units; a transaction engine to process the transactionlogs to determine to perform recovery for a subset of the set of thestorage units and to update a hash table to lock the subset of thestorage units, wherein the control engine is further to resume hostaccess requests before recovery is complete for the subset of the set ofthe storage units.
 2. The system of claim 1, the second node furthercomprising: a recovery engine to: initiate a plurality of recoveryoperations for the storage units from the subset, wherein each of therecovery operations is associated with one of a plurality of meta-data;for each of the recovery operations, determine that the respectivemeta-data is stored in a non-volatile memory; and based on eachdetermination, create, for each respective meta-data, a temporarymeta-data data structure, wherein the temporary meta-data data structureincludes an associated recovery transaction, wherein the temporarymeta-data data structure are created before the host access requests areresumed.
 3. The system of claim 2, further comprising: a page in engineto page in the respective meta-data to a volatile memory from thenon-volatile memory that is associated with the temporary meta-data datastructures using a background process.
 4. The system of claim 1, whereinthe second node further comprises: a recovery engine to: initiate arecovery operation for a first one of the storage units from the subset;determine that an exception table for the first one storage unit thatincludes mapping information is stored on a non-volatile memory; andbased on the determination, create a temporary exception table for theone storage unit and include an associated recovery transaction beforethe host access requests are resumed.
 5. The system of claim 4, whereinthe second node further comprises: a page in engine to page in themapping information to volatile memory from the non-volatile memory thatis associated with the temporary exception table after the host accessrequests are resumed.
 6. The system of claim 5, wherein a third one ofthe nodes includes a copy of the write information and wherein therecovery engine is further to: copy a first portion of the writeinformation on the third node to the second node based on thetransaction logs.
 7. The system of claim 5, wherein the control engineis further to: receive a host request for the first one storage unit;determine that the first one storage unit is locked based on the hashtable; prioritize performance of the associated recovery transaction;and unlock the first one storage unit for the host request.
 8. Thesystem of claim 7, wherein the control engine is further to: pause hostaccess requests for the first one storage unit while the associatedrecovery transaction is performed.
 9. A method comprising: controlling,at a first node of a plurality of nodes that are used to managetransactions to and from a plurality of storage units, a set of thestorage units, wherein the first node includes write information for theset of the storage units and maintains corresponding transaction logsthat include details of the transactions, wherein a second node of thenodes includes a copy of the transaction logs, assuming control, by thesecond node, of the set of the storage units; blocking host accessrequests to the set of storage units; determining to perform recoveryfor a subset of the set of the storage units based on the copy of thetransaction logs; creating a temporary exception table for one of thestorage units from the subset based on a determination that the onestorage unit is associated with one of a plurality of meta-data that isnot stored in a main memory of the second node; and resuming the hostaccess requests for the set of storage units after creating thetemporary meta-data.
 10. The method of claim 9, further comprising:updating a data structure to lock the subset of the storage units basedon the copy of the transaction logs.
 11. The method of claim 10, whereinthe one meta-data is an exception table and the temporary meta-data is atemporary exception table and wherein the temporary exception table iscreated to include at least one associated recovery transaction, themethod further comprising: paging in the one exception table to the mainmemory of the second node after resumption of the host access requests;and performing the associated recovery transaction after resumption ofthe host access requests; and updating the data structure to removeindication that the first storage unit has the associated recoverytransaction.
 12. The method of claim 10, wherein the temporary meta-datais created to include an associated recovery transaction, the methodfurther comprising: receiving a host request for the one storage unit;determining, based on the data structure, that the one storage unit islocked; prioritizing paging in the one meta-data to the main memory ofthe second node and performing the associated recovery transaction; andupdating the data structure to unlock the one storage unit.
 13. Themethod of claim 9, wherein the host access requests are resumed beforerecovery is complete for the subset of the set of storage units.
 14. Anon-transitory machine-readable storage medium storing instructionsthat, if executed by a physical processing element of a computingdevice, cause the computing device to: assume control, in a system wherea plurality of nodes are used to manage transactions to and from aplurality of storage units, of a set of storage units previouslycontrolled by a first one of the nodes, wherein the first node includeswrite information for the set of the storage units and maintainscorresponding transaction logs that include details of the transactions;wherein the device includes a copy of the transaction logs; block hostaccess requests to the set of storage units; determine to performrecovery for a subset of the set of the storage units based on the copyof the transaction logs; update a data structure to lock the subset ofthe storage units based on the copy of the transaction logs; create atemporary exception table for one of the storage units from the subsetbased on a determination that the one storage unit is associated with anexception table that is not stored in a main memory of the device,wherein the temporary exception table includes an associated recoverytransaction; and resume the host access requests for the set of storageunits after creating the temporary exception table.
 15. Thenon-transitory machine-readable storage medium of claim 13, furthercomprising instructions that, if executed by the physical processingelement, cause the computing device to: page in the exception table tothe main memory of the device after resumption of the host accessrequests; and perform the associated recovery transaction afterresumption of the host access requests; and unlock the one storage unit.16. The non-transitory machine-readable storage medium of claim 14,further comprising instructions that, if executed by the physicalprocessing element, cause the computing device to: receive a hostrequest for the one storage unit; determine, based on the datastructure, that the one storage unit is locked; prioritize paging in theexception table to the main memory of the device; perform the associatedrecovery transaction, wherein the recovery transaction includes causinga portion of a copy of the write information on a second one of thenodes to be copied to the device; and update the data structure tounlock the one storage unit.
 17. The non-transitory machine-readablestorage medium of claim 14, wherein the host access requests are resumedbefore recovery is complete for the subset of the set of storage units.